Method for filtering normal medical image, method for interpreting medical image, and computing device implementing the methods

ABSTRACT

A method of reading a medical image by a computing device operated by at least one processor is provided. The method includes obtaining an abnormality score of the input image using an abnormality prediction model, filtering the input image so as not to be subsequently analyzed when the abnormality score is less than or equal to a cut-off score based on the cut-off score which makes a specific reading sensitivity; and obtaining an analysis result of the input image using a classification model that distinguishes the input image into classification classes when the abnormality score is greater than the cut-off score.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a Continuation Application of U.S. Pat. ApplicationNo. 17/077,142 filed on Oct. 22, 2020, which claims priority to and thebenefit of Korean Patent Application No. 10-2020-0072257 filed in theKorean Intellectual Property Office on Jun. 15, 2020, the entirecontents of which are incorporated herein by reference.

BACKGROUND A Field

The present disclosure relates to an artificial intelligence-basedmedical image reading technology.

B Description of the Related Art

In a medical field, various products utilizing an artificialintelligence (AI) technology has been developing, and a diagnosisassistance system implemented with the AI-based medical image readingtechnology is a representative example thereof. The AI-based medicalimage reading technology can analyze the entire medical image with an AIalgorithm and provide an abnormal lesion visually. A specialized doctorfor image reading (hereinafter, referred to as a “reader”) can beprovided with an analysis result of the medical image from the diagnosisassistance system and read the medical image with reference thereto.

The reader can check a reading result provided by the diagnosisassistance system and medical records of a patient in a worklist, andcan change the image reading order by way of worklist sorting based onspecific criteria (e.g., emergency, abnormality, etc.). Using thefunction of worklist sorting, the reader can preferentially read animage required to be read urgently or an image where an abnormality isdetected, rather than an image analyzed as normal. However, since thefunction worklist sorting is only to change the reading order of theimages already included in the worklist, reading a normal image shouldbe done in the end. Therefore, workload of the reader does not change.

In addition, although the reading level of the diagnosis assistancesystem has been increasing, reading sensitivity of the diagnosisassistance system is not very high due to the trade-off between thesensitivity and specificity.

SUMMARY

An embodiment is to provide a method of calculating an abnormality scoreof an input image using an abnormality prediction model, and filteringthe input image whose abnormality score is less than or equal to acut-off score into strong normal, based on the cut-off score whichdetermines a specific sensitivity.

Another embodiment is to provide a method of creating a worklist forperforming subsequent classification analysis on images that are notstrong normal and excluding images classified into strong normal fromthe worklist, through a two-stage analysis including strong normalfiltering.

Yet another embodiment is to provide a method of cutting off multiplediseases whose disease prediction difficulties are different with thesame sensitivity, and a calibration method for performing the same.

According to an embodiment, a method of reading a medical image by acomputing device operated by at least one processor is provided. Themethod includes obtaining an abnormality score of the input image usingan abnormality prediction model, filtering the input image so as not tobe subsequently analyzed when the abnormality score is less than orequal to a cut-off score based on the cut-off score which makes aspecific reading sensitivity, and obtaining an analysis result of theinput image using a classification model that distinguishes the inputimage into classification classes when the abnormality score is greaterthan the cut-off score.

The input image whose abnormality score is less than or equal to thecut-off score may be classified into strong normal.

The classification model may include an artificial intelligence modelthat has learned a task of distinguishing the input image into weaknormal or abnormal.

The method may further include adding the analysis result to a readingworklist, and the input image whose abnormality score is less than orequal to the cut-off score may be not added to the reading worklist.

In a case of the input image whose abnormality score is less than orequal to the cutoff score, a filtering result may be added to a separatereport from the reading worklist.

Obtaining the abnormality score may include, when obtaining diseaseprediction scores for different diseases from the abnormality predictionmodel, aggregating the disease prediction scores to determine theabnormality score.

Obtaining the abnormality score may include, obtaining calibrateddisease prediction scores based on calibration that converts a cut-offscore for each disease, which makes the specific reading sensitivity,into the cut-off score, and determining a maximum value among thecalibrated disease prediction scores as the abnormality score.

In a case where each of disease prediction scores for different diseasesis obtained as an abnormality score for each disease from theabnormality prediction model, filtering the input image may includecalculating a cut-off score for each disease which makes the specificreading sensitivity, and filtering the input image when the abnormalityscore for each disease is less than or equal to the cut-off score for acorresponding disease, for all of the different diseases.

The abnormality prediction model may include a feature extraction modeltrained to output a feature of the input image, and at least one diseaseprediction head model trained to predict at least one disease based onfeatures output from the feature extraction model.

The abnormality prediction model may have a sensitivity between 90% and100%.

According to another embodiment, a method of reading a medical image bya computing device operated by at least one processor is provided. Themethod includes setting a cut-off score that makes a specific readingsensitivity for an abnormality prediction model that outputs anabnormality score of a medical image; obtaining an abnormality score ofan input image using the abnormality prediction model; and when theabnormality score of the input image is less than or equal to thecut-off score, classifying the input image into strong normal andfiltering the input image so as not to be subsequently analyzed.

The method may further include obtaining an analysis result of the inputimage using a classification model that distinguishes between weaknormal and abnormal when the abnormality score of the input image isgreater than the cut-off score.

The method may further include adding the analysis result to a readingworklist, and the input image classified into strong normal may be notadded to the reading worklist.

Obtaining the abnormality score may include, when obtaining diseaseprediction scores for different diseases from the abnormality predictionmodel, calibrating the disease prediction scores so as to provide thespecific reading sensitivity for the different diseases at the cut-offscore, and determining a maximum value among the calibrated diseaseprediction scores as the abnormality score of the input image.

In a case where each of disease prediction scores for different diseasesfrom the abnormality prediction model is obtained as an abnormalityscore for each disease, filtering the input image may includecalculating a cut-off score for each disease which makes the specificreading sensitivity, and filtering the input image when the abnormalityscore for each disease is less than or equal to the cut-off score for acorresponding disease, for all of the different diseases.

According to yet another embodiment, a method of reading a medical imageby a computing device operated by at least one processor is provided.The method includes inputting an input image to an abnormalityprediction model that outputs an abnormality score of a medical image,obtaining disease prediction scores for a plurality of diseases from theabnormality prediction model, calibrating the disease prediction scoresso as to provide a same reading sensitivity for the plurality ofdiseases at a same cut-off score, determining a maximum value among thecalibrated disease prediction scores as an abnormality score of theinput image, and when the abnormality score of the input image is lessthan or equal to the cut-off score, classifying the input image intostrong normal and filtering the input image so as not to be subsequentlyanalyzed. The abnormality prediction model may include a featureextraction model trained to output a feature of the input image, and aplurality of disease prediction head models that are trained to predictthe plurality of diseases based on features output from the featureextraction model.

The input image may include a chest x-ray image, and the plurality ofdiseases may include at least two of consolidation, nodule, andpneumothorax.

The reading sensitivity may have a value between 90 and 100%.

The method may further include obtaining an analysis result of the inputimage using a classification model that distinguishes between weaknormal and abnormal when the abnormality score of the input image isgreater than the cut-off score, and adding the analysis result to areading worklist. The input image classified into strong normal may benot added to the reading worklist.

According to still another embodiment, a computing device including amemory and a processor is provided. The memory stores an abnormalityprediction model trained to output an abnormality score of an inputimage, and a classification model trained to distinguish the input imageinto weak normal and abnormal. The processor obtains an abnormalityscore of the input image using the abnormality prediction model,classifies the input image into strong normal and filters the inputimage so as not to be subsequently analyzed when the abnormality scoreof the input image is less than or equal to a cut-off score, and obtainsan analysis result of the input image using the classification modelthat distinguishes the input image into classification classes when theabnormality score of the input image is greater than the cut-off score.

The abnormality prediction model may include a feature extraction modeltrained to output a feature of the input image, and at least one diseaseprediction head model trained to predict at least one disease based onfeatures output from the feature extraction model.

In general image medical diagnosis, most cases are normal cases wherethere is nothing abnormal. According to some embodiments, since strongnormal images corresponding to definite normal cases are excluded from aworklist, a workload of a reader can be significantly reduced.

According to some embodiments, a computing device does not perform asubsequent analysis on images classified into strong normal andintensively analyzes only images other than the strong normal images,thereby improving computing efficiency.

According to some embodiment, since the input image is analyzed using anartificial intelligence model that has learned a difficult task ofdistinguishing between weak normal and abnormal, it is possible toimprove the performance of distinguishing between weak normal andabnormal.

According to some embodiments, it is possible to filter strong normalimages with a high sensitivity, and particularly, even if thedistribution of abnormality scores of multiple diseases is differentaccording to a difference in abnormality prediction difficulties, strongnormal images can be filtered with a single cut-off score.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a drawing illustrating learning of an abnormality predictionmodel and a classification model according to an embodiment.

FIG. 2 is a drawing schematically showing a two-stage analysis includingstrong normal filtering according to an embodiment.

FIG. 3A and FIG. 3B show an example of an abnormality prediction modelaccording to an embodiment.

FIG. 4 and FIG. 5 each are a flowchart of a strong normal filteringmethod according to an embodiment.

FIG. 6A and FIG. 6B are drawings schematically illustrating a merit of atwo-stage analysis including strong normal filtering according to anembodiment.

FIG. 7 is a flowchart of a medical image reading method according to anembodiment.

FIG. 8 is a configuration diagram of a computing device according to anembodiment.

DETAILED DESCRIPTION

In the following detailed description, exemplary embodiments of thepresent disclosure will be described in detail with reference to theaccompanying drawings so that those of ordinary skill in the art mayeasily implement the present disclosure. However, the present disclosuremay be implemented in various different forms and is not limited to theembodiments described herein. Accordingly, the drawings and descriptionare to be regarded as illustrative in nature and not restrictive. Likereference numerals designate like elements throughout the specification.

As used herein, unless explicitly described to the contrary, the word“comprise”, “include” or “have”, and variations such as “comprises”,“comprising”, “includes”, “including”, “has” or “having” will beunderstood to imply the inclusion of stated elements but not theexclusion of any other elements. In addition, the term “unit”, “-er”,“-or” or “module” described in the specification mean a unit forprocessing at least one function and operation, and may be implementedby hardware components or software components, and combinations thereof.

As used herein, a task refers to an assignment to be solved throughmachine learning or a work to be done through machine learning. Forexample, in a case of performing recognition, classification, andprediction from a medical image, each of the recognition,classification, and prediction may correspond to an individual task. Anartificial intelligence model of the present disclosure is a model forlearning at least one task, and may be implemented as software or aprogram to be executed on a computing device. The program is stored in astorage medium (non-transitory storage media) and includes instructionsfor executing operations of the present disclosure by a processor. Theprogram may be downloaded via a network, or sold as a product.

The present disclosure may be applied to medical images of various areasphotographed with various modalities. For example, the modalities ofmedical images may be X-ray, magnetic resonance imaging (MRI),ultrasound, computed tomography (CT), mammography (MMG), or digitalbreast tomosynthesis (DBT). In the specification, a chest X-ray imagemay be described as an example.

A general diagnosis assistance system analyzes an input image as normalor abnormal based on a cut-off score. The cut-off score is determined byadjusting the trade-off between sensitivity and specificity. If thesensitivity is set very high, false positives increase and thespecificity decreases, resulting in increased user fatigue. Therefore,the existing diagnosis assistance systems set the reading sensitivitynot very high so that the images analyzed as normal require a doctor’sreading. However, since most cases are normal cases having noabnormality, it is required to improve the efficiency of the readingwork.

In order to solve this issue, the present disclosure filters the imagesdetermined that is definitely normal so as not to require the doctor’sreading. Such non-suspicious and definite normal is referred to as“strong normal”. Among the normal cases, a case that is not strongnormal and thus requires the doctor’s reading is referred to as a “weaknormal”.

FIG. 1 is a diagram for explaining learning of an abnormality predictionmodel and a classification model according to an embodiment, and FIG. 2is a diagram schematically showing a two-stage analysis including strongnormal filtering according to an embodiment.

Referring to FIG. 1 , a computing device 10 operated by at least oneprocessor may train an abnormality prediction model 100, which is anartificial intelligence model, using at least some of training data 20.The abnormality prediction model 100 may learn about a task ofpredicting an abnormality for features of a medical image and outputtingthe predicted result as an abnormality score. Here, the abnormalityprediction model 100 is used to filter out a definite normal image(hereinafter referred to as a “strong normal” image) among input images,and has a high reading sensitivity. For example, the abnormalityprediction model 100 may have a very high sensitivity between 90% and100%. Hereinafter, it is described that the abnormality prediction model100 has an ultrahigh sensitivity of 99%.

The computing device 10 may train the abnormality prediction model 100by assigning a weight to abnormal images among the training data 20. Thecomputing device 10 may train the abnormality prediction model 100 byassigning the weight to the abnormal images that are difficult to beclassified, and iteratively train the abnormality prediction model 100on the abnormal images that are difficult to be classified. Thecomputing device 10 may train the abnormality prediction model 100 byadjusting the abnormal images to maintain a specific ratio in anobjective function.

The computing device 10 may train a classification model 200, which isan artificial intelligence model, using at least some of the trainingdata 20. The classification model 200 may learn a classification taskusing the training data annotated with weak normal and abnormal otherthan strong normal. That is, the classification model 200 mayintensively learn a difficult task of distinguishing the input imageinto weak normal or abnormal. The classification model 200 may beimplemented with various neural networks that can classify inputfeatures into classification classes. Here, “weak normal” means a casethat can be suspicious as malignant but is normal, and is used todistinguish normal into strong normal and normal that is not strongnormal.

Referring to FIG. 2 , a computing device 10 performs a two-stageanalysis including strong normal filtering using the learned abnormalityprediction model 100 and the learned classification model 200.

First, the computing device 10 may calculate an abnormality score of aninput image using the learned abnormality prediction model 100, and mayclassify and filter the input image whose abnormality score is less thanor equal to a cut-off score (e.g., 0.05) into strong normal, based onthe cut-off score which makes (or sets) a specific reading sensitivity(e.g., 99%). In this way, the computing device 10 may classify inputimages into strong normal images and the remaining images, and filterout the strong normal images so that no subsequent classificationanalysis is performed on the strong normal images. Then, the computingdevice 10 may perform the subsequent analysis for classifying ordifferentiating the remaining images into classes other than the strongnormal class. The classes other than strong normal class may include,for example, weak normal and abnormal, and may be further subdivided.

The computing device 10 classifies the remaining unfiltered images intothe classes such as weak normal or abnormal, using the learnedclassification model 200. The images with the classification resultother than strong normal, such as weak normal or abnormal, are added toa reading worklist (hereinafter, referred to as a “worklist”).Thereafter, when a reader selects an input image from the worklist, theanalysis result (disease position or disease prediction score, etc.)obtained by using the classification model 200 may be visually displayedon the input image. The visual display method may be selected from amongvarious methods including secondary capture.

The computing device 10 filters and classifies the input imagesaccording to the abnormality score, as in Table 1. The image classifiedinto strong normal is not further analyzed and is excluded from theworklist. The image that is not filtered out and is analyzed as normal(weak normal) is not excluded from the worklist and is required to bechecked by the reader. For the image analyzed as abnormal, a heatmapvisually indicating a position or predicted value of an abnormal lesionmay be displayed.

TABLE 1 Abnormality Score Case Action [0, 0.05) Strong normal Excludedfrom worklist [0.05,0.15) Normal/weak normal Not excluded from worklist,and Required to be checked by reader [0.15, 1.0] Abnormal Abnormalheatmap shown

In this way, the computing device 10 may exclude the strong normalimages from the worklist by filtering them through the two-stageanalysis including strong normal filtering based on the abnormalityscore and the subsequent analysis, and add only the remaining imagesincluding the results of the subsequent analysis to the worklist. Sincethe images filtered as strong normal have a very low probability ofabnormal lesions, there is no need to perform the subsequent analysisfor them and to add them to the worklist that is required to be read bythe reader unlike the images that are not strong normal. On the otherhand, the analysis result of the images filtered as strong normal may beprepared as a report in a different form than the worklist.

The worklist is a list of images that are required to be read by areader in a reading procedure, and, in a broad sense, may mean a list ofvarious tasks including medical actions related to patients in a medicalinstitution. Because the worklist merely includes images classified intonon-strong normal cases (e.g., weak normal and abnormal cases),unnecessary reading work for strong normal cases that are clearly normalmay be reduced. Since a ratio of the strong normal cases variesdepending on an image modality and a type of disease, the reductionratio of the reading workload may be different. In the normal cases thatoccupy the majority, the workload of the reader can be significantlyreduced because the strong normal images are excluded from the worklist.When a specific verification set is checked after a cut-off score of0.05, in which the sensitivity for the abnormal prediction model 100becomes 99%, is set, more than half of the normal images are filtered asstrong normal so that more than half of the normal images may beexcluded from the worklist.

Although a task of training the abnormal prediction model 100, a task offiltering strong normal cases using the abnormality prediction model100, a task of training the classification model 200, and a classifyingor distinguishing task using the classification model 200 may beimplemented over a plurality of computing devices in a distributedmanner, it is assumed that the computing device 10 performs operationsof the present disclosure for convenience of description. For example,after a specific computing device trains the abnormality predictionmodel 100 and/or the classification model 200, and the learnedabnormality prediction model 100 and the classification model 200 may beinstalled to computing devices positioned at a hospital, the imagereading based on the two-stage analysis may be performed. Alternatively,after the learned abnormality prediction model 100 and classificationmodel 200 may be installed to a server device, the computing devicespositioned in the hospital may transmit an image to the server device,receive the analysis result of the image from the server device, andthen add the analysis result to the worklist.

FIG. 3A and FIG. 3B show an example of an abnormality prediction modelaccording to an embodiment.

Referring to FIG. 3A and FIG. 3B, an abnormality prediction model 100 aor 100 b may be designed as a single-headed network or a multi-headednetwork. For example, the abnormality prediction model 100 a or 100 bmay include a feature extractor for extracting a feature of an inputimage, and one abnormality/disease prediction head model.

Referring to FIG. 3A, the abnormality prediction model 100 a designed asthe single-head network may include a feature extraction model 110 andan abnormality prediction head model 130. The feature extraction model110 is a neural network model trained to extract a feature for detectinga lesion from an input image, and outputs the feature of the inputimage. The abnormality prediction head model 130 is a neural networkmodel trained to predict an abnormality probability for the featuresoutput from the feature extraction model 110, and outputs a predictionresult as an abnormality score.

When obtaining the abnormality score of the input image using thelearned abnormality prediction model 100 a, a computing device 10 mayclassify and filter the input image whose abnormality score is less thanor equal to a cut-off score into strong normal, based on the cut-offscore for making a desired reading sensitivity (e.g., 99%).

Referring to FIG. 3B, the abnormality prediction model 100 b designed asthe multi-head network may include a feature extraction model 110 and aplurality of disease prediction head models 140-1, 140-2, ..., 140-n.Each disease prediction head model is a neural network model trained topredict a corresponding disease, and outputs the prediction result as acorresponding disease prediction score. Although the disease predictionscore may correspond to the abnormality score, a score output from thedisease prediction head model may be called the disease prediction scorein order to distinguish it from the abnormality score calculated bycombining the disease prediction scores.

The plurality of disease prediction head models 140-1 to 140-n) may beconfigured in parallel according to types of diseases (lesions orfindings) that can be analyzed in an image photographed with a specificmodality. For example, when the input image is a chest X-ray image, theabnormality prediction model 100 may include the disease prediction headmodels that independently predict consolidation, nodule, andpneumothorax, respectively.

In a case of the abnormality prediction model 100 b, each of theplurality of disease prediction head models 140-1 to 140-n) isindependently trained based on learning data related to a correspondingdisease. In this case, a prediction difficulty of each disease may bedifferent, and an amount of training data related to each disease may bedifferent. Therefore, since the distribution of disease predictionscores output from each disease prediction head model is different, thedisease prediction score that becomes the desired reading sensitivity(e.g., 99%) may be different for each disease. If the strong normalimages and the remaining images are classified based on the same cut-offscore (e.g., 0.05), the reading sensitivity may vary for each disease.For example, a lesion that is easy to detect may be sensitivelydetected, and a lesion that is difficult to detect may be detected lesssensitively. In order to solve this issue, a method of filtering imagesso as to allow the computing device 10 to provide the same readingsensitivity (e.g., 99%) regardless of a disease type is described below.

FIG. 4 and FIG. 5 each are a flowchart of a strong normal filteringmethod according to an embodiment.

Referring to FIG. 4 , a computing device 10 obtains disease predictionscores (e.g., a consolidation score, a nodule score, and a pneumothoraxscore) of an input image that are output from disease prediction headmodels of an abnormality prediction model 100 b, respectively (S110).

The computing device 10 sets cut-off scores C1, C2, and C3 which make aspecific reading sensitivity (e.g., 99%) for each disease, and comparesthe cut-off score for each disease with the disease prediction score(S120). Here, the cut-off score for each disease may not be the same. Ifthe disease prediction score is less than the cut-off score, the inputimage is filtered as strong normal.

The computing device 10 determines whether the input image is strongnormal or non-strong normal for each disease based on the comparisonresult, and determines whether the input image is classified into strongnormal for all of the plurality of diseases (S130). The input image withthe disease prediction score less than or equal to the cut-off score isclassified into strong normal, and the input image with the diseaseprediction score greater than the cut-off score is classified intonon-strong normal.

The computing device 10 filters the input image so as not to perform asubsequent classification analysis when the input image is classifiedinto strong normal for all of the plurality of diseases, and forwardsthe input image to a subsequent classification stage when the inputimage is not classified into strong normal for all of the plurality ofdiseases (S140).

That is, the computing device 10 finally classifies the input image asstrong normal when the input image is classified into strong normal forall of the plurality of diseases, and determines that the input image isnot strong normal when the input image is not classified into strongnormal for all of the plurality of diseases. For example, when all ofthe consolidation score, nodule score and pneumothorax score predictedfor the input image are less than or equal to the cut-off score of thecorresponding disease, the input image is classified into strong normal.The input image classified into strong normal is not added to aworklist, unlike an image on which the subsequent classificationanalysis is to be performed. Instead, the computing device 10 may createa separate report, which is distinguished from the worklist, for theinput image classified into strong normal.

Referring to FIG. 5 , a computing device 10 obtains disease predictionscores (e.g., a consolidation score, a nodule score, and a pneumothoraxscore) of an input image that are output from disease prediction headmodels of an abnormality prediction model 100 b, respectively (S210).

The computing device 10 calibrates each disease prediction score so asto make a specific reading sensitivity (e.g., 99%) at a specific cut-offscore (e.g., 0.05) for all diseases (S220).

The computing device 10 determines a maximum value among the calibrateddisease prediction scores as an abnormality score (S230).

The computing device 10 compares a specific cut-off score of 0.05 withthe abnormality score, and determines whether the input image is strongnormal or non-strong normal based on the comparison result (S240).

The computing device 10 filters the input image so as not to perform asubsequent classification analysis when the input image is classifiedinto strong normal, and forwards the input image to a subsequentclassification stage when the input image is not classified into strongnormal (S250). The computing device 10 may create a separate report forthe input image classified into strong normal without adding theanalysis result to the worklist.

The computing device 10 may calibrate each disease prediction score asfollows. The calibrated disease prediction scores may be calculatedbased on calibration that converts the cut-off score for each disease,which makes the specific reading sensitivity, into the same cut-offscore. As a result, even if the input image is classified into strongnormal at a single cut-off score, the same reading sensitivity for alldiseases may be provided.

For example, assuming that a cut-off score for making a sensitivity of99% in a pneumothorax prediction head model is c1, a pneumothorax scorey1 may be converted into a pneumothorax score y1′ through Equation 1, inorder to shift the cut-off score c1 to a specific score (e.g., 0.05).

$\begin{matrix}{y1^{\prime} = \frac{y \ast 0.05}{c1},\quad if\mspace{6mu} y1 \leq c1} \\{y1^{\prime} = \frac{\left( {y1 - c1} \right) \ast \left( {1 - 0.05} \right)}{1 - c1} + 0.05,\quad if\mspace{6mu} y1 > c1}\end{matrix}$

Similarly, assuming that a cut-off score for making a sensitivity of 99%in a consolidation prediction head model is c2, a consolidation score y2may be converted into a calibrated consolidation score y2′ as inEquation 1.

Assuming that a cut-off score for making a sensitivity of 99% in anodule prediction head model is c3, a nodule score y3 may be convertedinto a calibrated nodule score y3′ as in Equation 1.

The computing device 10 determines a maximum value among the calibrateddisease prediction scores y1′, y2′, and y3′ as an abnormality score, andcompares the cut-off score of 0.05 with the abnormality score. If theabnormality score is less than or equal to the cut-off score of 0.05,the input image is classified into strong normal.

FIG. 6A and FIG. 6B are drawings schematically illustrating a merit of atwo-stage analysis including strong normal filtering according to anembodiment.

Referring to FIG. 6A, a general classification model is trained todistinguish normal including strong normal and weak normal fromabnormal. Since distinguishing input features into weak normal andabnormal is difficult compared to distinguishing the input features intostrong normal and abnormal, the general classification model tends toperform learning more easily and does not properly learn a task fordistinguishing between weak normal and abnormal.

On the other hand, as shown in FIG. 6B, a computing device 10 can filterimages predicted as strong normal through strong normal filtering toexclude them from subsequent classification, and classify input featuresinto weak normal and abnormal through a classification model 200 fordistinguishing between weak normal and abnormal. The classificationmodel 200 may intensively learn a difficult task for distinguishingbetween weak normal and abnormal, and thus can have high classificationperformance.

The computing device 10 may add the analysis result on the input imageof the classification model 200 to the worklist. The analysis result mayinclude a detection result for at least one disease (e.g.,consolidation, nodule, pneumothorax, etc.) that can be analyzed inmodality.

FIG. 7 is a flowchart of a medical image reading method according to anembodiment.

Referring to FIG. 7 , a computing device 10 receives images to be read(S310).

The computing device 10 calculates an abnormality score of each inputimage using the learned abnormality prediction model 100 (S320). Theabnormality prediction model 100 may be an abnormality prediction model100 a configured with a single-head network or an abnormality predictionmodel 100 b configured with a multi-head network. The computing device10 may calibrate a plurality of disease prediction scores output fromabnormality prediction models 100 b so as to make the same readingsensitivity at the same cut-off score for all diseases. Further, thecomputing device 10 may determine a maximum value among the calibrateddisease prediction scores as the abnormality score.

The computing device 10 filters an input image whose abnormality scoreis less than or equal to the cut-off score as strong normal, based on acut-off score that makes a specific reading sensitivity, and determinesan input image whose abnormality score is greater than the cut-off scoreas a subsequent analysis target (S330).

The computing device 10 performs an analysis of distinguishing the inputimages determined as the subsequent analysis target into classificationclasses (e.g., weak normal and abnormal) using a classification model200 (S340).

The computing device 10 adds the analysis result to a worklist that isrequired to be read by a reader (S350).

The computing device 10 may create a separate report for the input imagefiltered as strong normal (S360).

FIG. 8 is a configuration diagram of a computing device according to anembodiment.

Referring to FIG. 8 , a computing device 10 may include one or moreprocessors 11, a memory 13 on which a computer program to be executed bythe processor 11 is loaded, a storage 15 which stores the program andvarious data, a communication interface 17, and a bus 19 for connectingthem. In addition, the computing device 10 may further include variouselements. The program may include instructions which make the processor11 to perform methods or operations according to various embodiments ofthe present disclosure when loaded on the memory 13. In other words, theprocessor 11 may perform methods or operations according to variousembodiments of the present disclosure by executing the instructions. Theprogram includes a series of computer-readable instructions that aregrouped by function and is executed by the processor.

The processor 11 controls overall operation of each element of thecomputing device 10. The processor 11 may be configured to include atleast one of a central processing unit (CPU), a microprocessor unit(MPU), a microcontroller unit (MCU), a graphics processing unit (GPU),and any form of processor well known in the technical field of thepresent disclosure. Further, the processor 11 may perform computationfor at least one application or program to execute methods or operationsaccording to embodiments of the present disclosure.

The memory 13 stores various kinds of data, commands, and/orinformation. To execute methods or operations according to variousembodiments of the present disclosure, the memory 13 may load one ormore programs from the storage 15. The memory 13 may be implemented as avolatile memory such as a random access memory (RAM), but the technicalscope of the present disclosure is not limited thereto.

The storage 15 may non-temporarily store the program. The storage 15 mayinclude a non-volatile memory, such as a read only memory (ROM), anerasable programmable ROM (EPROM), an electrically erasable programmableROM (EEPROM) or a flash memory, a hard disk, a removable disk, or anyform of computer-readable recording medium well known in the art towhich the present disclosure pertains.

The communication interface 17 supports wired or wireless Internetcommunication of the computing device10. Further, the communicationinterface 17 may support various communication methods as well asInternet communication. To this end, the communication interface 17 mayinclude a communication module well known in the technical field of thepresent disclosure.

The bus 19 provides a communication function between elements of thecomputing device 200. The bus 19 may be implemented as various forms ofbuses, such as an address bus, a data bus, and a control bus.

The embodiments of the present invention described above are notimplemented through only the apparatus and the method, but may also beimplemented through a program that realizes functions corresponding tothe configuration of the embodiments of the present invention or arecording medium on which the program is recorded.

While this invention has been described in connection with what ispresently considered to be practical embodiments, it is to be understoodthat the invention is not limited to the disclosed embodiments, but, onthe contrary, is intended to cover various modifications and equivalentarrangements included within the spirit and scope of the appendedclaims.

What is claimed is:
 1. A method of reading a medical image by acomputing device operated by at least one processor, the methodcomprising: obtaining an abnormality score of an input image using anartificial intelligence model; classifying the input image as strongnormal, and excluding the input image from being subsequently analyzedwhen the abnormality score is less than or equal to a cut-off score; andobtaining an analysis result of the input image using the artificialintelligence model that distinguishes the input image into classesincluding weak normal or abnormal when the abnormality score is greaterthan the cut-off score.
 2. The method of claim 1, further comprisingadding the input image having the analysis result to a worklist.
 3. Themethod of claim 2, wherein the input image in the worklist is requiredto be checked by reader.
 4. The method of claim 2, further comprisingwhen a specific image in the worklist is selected, displayingcorresponding analysis result of the specific image.
 5. The method ofclaim 4, wherein the corresponding analysis result is visually displayedas a secondary capture image.
 6. The method of claim 4, wherein when thespecific image analyzed as abnormal is selected, displaying a heatmapvisually indicating a position or predicted value of an abnormal lesion.7. The method of claim 1, wherein the input image being strong normal isexcluded from a worklist, and provided as a different form than theworklist.
 8. A computing device comprising: a memory; and at least oneprocessor that executes instructions of a program loaded in the memory,wherein the processor obtains an abnormality score of an input imageusing an artificial intelligence model; classifies the input image asstrong normal, and excludes the input image from being subsequentlyanalyzed when the abnormality score is less than or equal to a cut-offscore; and obtains an analysis result of the input image using theartificial intelligence model that distinguishes the input image intoclasses including weak normal or abnormal when the abnormality score isgreater than the cut-off score.
 9. The computing device of claim 8,wherein the processor adds the input image having the analysis result toa worklist.
 10. The computing device of claim 9, wherein the input imagein the worklist is required to be checked by reader.
 11. The computingdevice of claim 9, wherein the processor provides corresponding analysisresult of specific image, when the specific image in the worklist isselected.
 12. The computing device of claim 11, wherein thecorresponding analysis result is visually displayed as a secondarycapture image.
 13. The computing device of claim 11, wherein theprocessor provides a heatmap visually indicating a position or predictedvalue of an abnormal lesion, when the specific image analyzed asabnormal is selected.
 14. The computing device of claim 8, wherein theinput image being strong normal is excluded from a worklist, andprovided as a different form than the worklist.